Applying multiagent reinforcement learning to distributed function optimization problems

Consider a set of non-cooperative agents acting in an environment in which each agent attempts to maximize a private utility function. As each agent maximizes its private utility we desire a global ”world” utility function to in turn be maximized. The inverse problem induced from this situation is the following: How does each agent choose his move so that while he optimizes his private utility, the world utility is optimized as well? This problem has been considered by the theory of COllective INtelligence (COIN) [6]. This paper focuses on a method for improving a class of search algorithms using Intelligent RL based learning players engaging in a non-cooperative game. The players or ”coordinates” use a more traditional AI approach to learning in a system in which largely the state of the world is not known, and effects of each agent’s actions on the world are not known a priori, and must be discovered. This is essentially a search problem through the possible policies of each coordinate of the underlying system. Search algorithms such as Simulated Annealing, Swarm Intelligence, and Genetic Algorithms, that tradeoff between exploration and exploitation are particularly useful in solving systems of this nautre, because they search directly in the space of RL policies, and try to select optimal policies which pay over time through simulation of indirect learning via optimization of the system; however, these algorithms all have one major drawback: They do not rely on the previous experiences of players in the system to help determine which actions the players should take next. This major drawback leads Wolpert and Tumer to develop a new method for improving search algorithms, Intelligent Coordinates[7]. Intelligent Coordinates use RL to remember and ”learn” how to better solve the system, by applying traditional RL techniques to the exploration stage of these search algorithms. The algorithm improves the binpacking simulation by an order of magnitude, and increases linearly with the number of items added. I present a more detailed explanation of the Intelligent Coordinates algorithm, and discuss two implementations of the bin-packing simulation that I have constructed in Java, one that uses Simulated Annealing to solve, and one that uses Intelligent Coordinates. Further, I contribute an efficient method for processing time-weighted sums, which are needed in the simulation. I also formalize a method for calculating the time-weighted probabilities of picking a particular bin B in the simulation. These probabilities are used to ”mask” the original SA distribution to achieve the more relevant IC probability distribution. I also present some original experiments in the bin-packing simulation that were not present in the original paper[7]. 1 University of Southern California, Los Angeles, email:[email protected]